A POTENTIAL THEORETIC CHARACTERIZATION OF COMPACTNESS 

OF THE 9-NEUMANN PROBLEM 
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ABSTRACT. We give a potential theoretic characterization for compactness of the 
3-Neumann problem on smooth bounded pseudoconvex domains in C". 



Let fl be a domain in C n with C°°-smooth boundary. The domain O is said 
to be pseudoconvex if the Levi form of O, the restriction of the complex Hes- 
sian of a defining function onto complex tangent space, is positive semi-definite 
on the boundary, bQ, of O. On bounded pseudoconvex domains, Hormander 
|Hor65] showed that the 9-Neumann operator on O, the solution operator for □ 
is a bounded operator on L 2 ^ (O) (here □ = 99 + 99 and 9 is the Hilbert space 

adjoint of 9). We refer the reader to ||CS01l IStrlO] for more information about the 
9-Neumann problem. 

Compactness of the 9-Neumann operator is important to study as it is weaker 
than global regularity [KN65] and it interacts with the boundary geometry. For 
example, even though the general case is still open, in same cases it is know that 
existence of an analytic disc in the boundary is an obstruction for compactness 
of the 9-Neumann problem (see, for exa mple, IFS981 iFSOll |§ah06[ |§S06l ISlrTolH . 
Recently, Straube and Munasinghe [MS07] (see also [Mun06]|) studied compactness 
using geometric conditions involving short time flows on the boundary (this was 
done in C 2 earlier by Straube llStrOl V g e lik and Straube (^509| (see also |Cel08| ) 
explored compactness in relation to the so called "compactness multipliers". 

Compactness of the 9-Neumann problem has been studied using some poten- 
tial theoretic conditions by Catlin |Cat84] using property (P) and later by McNeal 
[McN02] using property (P). In this paper we would like to give a new potential 
theoretic characterization for compactness of the 9-Neumann problem. We refer 
the reader to UStrlOl Proposition 4.2] for other equivalent conditions. We would 
like to note that a similar characterization has been done by Haslinger in [Has J. 

Let O be a smooth bounded domain in C M , K C bCl, and U be an open neigh- 
borhood of K. We denote the L 2 norm and Sobolev — 1 norm of a function / £ 
L 2 (Q) by ll/H and ||/||-i, respectively. Let I = ■ ■ ■ C N such that 

jl < i 2 < • • • < ip- Then we use the notation &z\ = dz^ A dz i2 A • • • A d-Zj and 
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dzi = dz[ x A dz~j 2 A • • • A dz\ . Define 



Co M (M )(^) = i E fi^Adzj-.fueC^iU) 

for < ^ < n. Define A^^IZ) as follows: for < p < n and 1 < ^ < n — 1 let us 
define 
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A (M) (U) = inf | 11 J 11 y 11 : / € Dom{d ) n C~ (m) (U),/ ^ 

and 

A ( , /0) (U) = inf | JM : / e (Kerd) 1 n C °^ 0) (U),/ ^ 

where (KerB) 1 - is the orthogonal complement of (Kerd) in L? x(O) (square inte- 
grable (p, 0)-forms on O). Notice that A( M ) (IT) < A( P/(? ) (V ) if V C IT. In this paper 
a finite type is meant in the sense of D'Angelo [D'A82] and infinite type point 
means a point that is not finite type. 

Theorem 1. Let O be a smooth bounded pseudoconvex domain in C n , n > 2 and < 
p < n, < q < n — 1, be given. Then the following are equivalent: 

(i) the d-Neumann operator N^ p ^ of CI is compact on L 2 s (O), 

(ii) for any compact set K C bCl and M > fere exzsfs an open neighborhood UofK 
such that A( p „)(li) > M, 

(iii) for any M > fere exz'sfs an open neighborhood U of the set of infinite type points 
in bCl such that \^„„\(U) > M. 

Remark 1. The definition of k/„ „\ is closely connected to the so-called compactness 
estimates (see dD in the proof of Theorem []} as well as Morey-Kohn-Hormander 
formula (see, for example, HCSOll Proposition 4.3.1] or UStrlOl Proposition 2.4]) and 
property (P) of Catlin. 

One can show that the Morey-Kohn-Hormander formula implies that for a smooth 
bounded pseudoconvex Q C C w , a non-positive function b £ C 2 (Q), and u G 
Dom (8) n Dora (3 ) n Cj } (O) we have 

E E l^ eb ^^ u J,jt^m dv ^ IM 2 + I|d~*«ll 2 - 

J,Kj,k=l Jn a/ 'i a/ 'k 

where u = Y!j k Y%=i u j,kK^ A dz^ A dz^ and the prime indicates that the sum 
is taken over strictly increasing (p,q — l)-tuples (J,K). If the domain O satisfies 
property (P) then one can choose b to be bounded from below by —1 and with ar- 
bitrarily large complex Hessian on the boundary of Cl. Then on a small neighbor- 
hood on the boundary the Hessian is still large. Hence A( p ^ (IT) will be arbitrarily 
large for a sufficiently small neighborhood U of bCl. 
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We would like to give a simple example below to show that one can use this 
characterization to show that, in some cases, compactness of the 9-Neumann prob- 
lem excludes analytic disks from the boundary. We do not claim any originality in 
this example as it is a special case of Catlin's result [FS01 , Proposition 1]. 

Example 1. Let O be a smooth bounded pseudoconvex domain in C 2 such that 

O C {z € C 2 : Im(z 2 ) < 0} and {z G C 2 : Im(z 2 ) = 0, |zi| 2 + |z 2 | 2 < 1} C bd. 

Claim: The 9-Neumann operator on O is not compact. 

Proof of the Claim: There exist positive numbers a\ < a 2 such that 

DixW]Cfln{zGC 2 : |zi| 2 + |z 2 | 2 < 1} C D 2 x W 2 
where D 1 = {z G C : |z| < 2/3}, D 2 = {z G C : |z| < 2}, and 



Wi 

W 2 



re' fl G C : < r < a lf -In I '3 < < -tt/3}, 



{z = re ie G C : < r < a 2 , -4tt/3 < 6 < tt/3}. 

Let <pj{z\,Z2) = f(zi)gj(z2)dz~i where / € C~(Di) and / ^ 0. Later on we will 

choose gy G C~({z G C : |z| < ;~ 2 }) so that <pj G Dom(d) n Dom(d*). There exists 
fl 3 > such that D x x W C O, where W = {z G C : Im(z) < 0, |z| < a 3 }, and 
<pj{z\, z 2 ) = for z G O \ Di x W and ; -2 < a.3. Then for y~ 2 < a$ we have 
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; 2 ,j 2 ) such that 



Let us choose real valued non-negative functions X) £ Q 
#y(— = ^;'(0 an d Pt(0 = 1 for I f I < jp. Since z~ 2 is not integrable on W\ n B(0, e 
for any e > 0, we can choose a positive real number ay so that 

' 4^(Z2) < / 

tj\ J Wins (o,i 



w 2 ns (0,1/7) F2 
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dV(z 2 ). 



Now we define gy(z 2 ) = #y (|z 2 | 2 ) Ty(z 2 )(z 2 — /a,-) 1 where Ty G C°°(C) such that 
1 for Im(z) < and Ty(z) = for Im(z) > ay/2. Then we have (pj G 



T; z 



C~ (01) (LIy) n Dom(9 ) where Lly = {z G C : |z| < 2" 1 x {z G C : |z| < ;" 2 } 

and 

S ||g;llL2(WinB(Q,l/j))- 

L 2 (W 2 nB(0,l/j)) 



9z 2 
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Hence, we constructed a sequence of (0, 1) -forms {cpj} such that <pj G C^ ^ (LTy) n 
Dora (9*) where 

oo 

K = {z G C 2 : |zx| < l/2,z 2 = 0} = f| LTy C 

7=1 

and ^-^txw — stays bounded as / — > oo. Hence, by Theorem [TJ the 9-Neumann 
operator on O is not compact. 

Proof of Theorem Q] 

Proof of Theorem]]] We will show the equivalences for < p < n and 1 < q < n — 1. 
The proof can be mimicked for the case q = using the following: compactness of 
No is equivalent to the following compactness estimate: for all e > there exists 
D e > such that 

||g|| 2 < e\\dg\\ 2 + DeWgf^ forge (Kerd^nDomfi) 

First let us prove that (i) implies (it). Assume that the 9-Neumann operator of 
O is compact, and there exist K C bCi and M > such that A^^(U) < M for all 
open neighborhoods U of K. We may assume that there exist sequences of open 
neighborhoods {U k } of K and nonzero (p, g)-forms {f k } such that 

i. u k+1 ^u k ,Kc nr=i u k c ba,f k G Dom(a*) n c~ (M) (i/ fc ), 

ii. Il/fcf = 1, and ||a/, c || 2 + ||aVfc|| 2 < Mforfc = 1,2,3,- • • 

Since K C DfcLi Ufc C fcO (hence K has measure in C w ) and f k G (? )(^fc)/ 
by passing to a subsequence if necessary, we may assume that — //|| 2 > 1/2. 
Compactness of the 9-Neumann operator is equivalent to the following so called 
compactness estimate (see [StrlO , Proposition 4.2] or [FS01 , Lemma 1]): for all e > 
there exists D £ > such that 

(1) \\g\\ 2 < e(||%|| 2 + ||9*gH 2 ) + DjgH 2 .! forg G Dora(9*) n Dora(9) 

Choose £ = Since Dora(9*) n v(lZjfc) C Dom(d ) n Dom(d) using (d) and 
ii. above we get 

(2) ll/fc-//|| 2 i> i ^>0for^I 

The imbedding from L 2 (Q) to W _1 (n) is compact and {/^} is abounded sequence 
in L 2 p^(D). Hence {f k } has a convergent subsequence in W7 1 %(£~2). This contra- 
dicts with ©. 

(zz) obviously implies (z'z'z) so we will skip this part. 

Next let us prove that (Hi) implies (i). Let K be the set of infinite type points in 

bCl and u G Dom(d ) n s(O). Assume that A( M )(14) > fc where {Iffc} is a 

sequence of open neighborhoods of K such that Ujt+i i-4 an d ^ C flfcLi ^fc C 
Let G CQ X, (LTfc) such that < ^ < 1 and ^ = 1 in a neighborhood of 
K. Define xp k = 1 — <p^- Notice that ip k is supported away from K. In following 
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estimates, Q and Q C/£ are general constants meaning that the constants depend on 
the subscripts only but they might change at each step. Away from K we have 
subelliptic estimates as bCl \ K is the set of finite type points (see HCat871 ). Hence, 
there exists s > for all £ > there exists D £ > such that 

||^fc w l| 2 < e ll t /'A: M lls + ^ell^fc^ll-l 

(3) < eC fc (||a(^w)|| 2 + ||9*(^ fc w)|| 2 ) + C fc , £ ||w||2 a 

< eC/ c ( 1 1 3tf 1 1 2 + ||9*w|| 2 + ||w|| 2 ) + Ca:,e II M II —i 

The first inequality follows because L 2 imbedds compactly into W s for s > 0. We 
used the compactness estimate for the second inequality If we use A( p „\ (U^) > k 
we get: 

1 — — * 

\Wk u \\ 2 < ^(\\d(<Pk u )\\ 2 + II 3 Ofc")|| 2 ) 

(4) < ^ ( 1 1 a w 1 1 2 + ||a*w|| 2 ) + D k \\fau\\ 2 

where fa = in a neighborhood of K, D k > 0, and fa = 1 in a neighborhood of the 
support of <pfc. Calculations that are similar to ones in © show that 

(5) II^A: M || 2 < £/ Q(||9w|| 2 + ||9*w|| 2 + ||w|| 2 ) + C^e'Hull 2 ,! 

By choosing e, e' > small enough and combining © and © we get the following 
estimate: for all k = 1, 2, 3, • • • there exists Mfc > such that 

(6) IMI 2 < ^(\\M\ 2 + ||a*u|| 2 + |H| 2 ) + M k \\u\\ 2 _ x for u e Dom(d*) n c^(n) 

We note that Dom(d*) n C?° JO) is dense in Dora (9* ) n Dora (9). Therefore, the 

above estimate © holds on Dom(d ) n Dom(d). That is, the 9-Neumann operator 
of O is compact on (p, ^)-forms for < p < n and 1 < q < n — 1. □ 
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